Image correction system, image correction method, and computer program product

ABSTRACT

This invention is concerning an image correction system that includes an image-capturing unit configured to acquire a captured image by capturing a subject; a tilt recognition unit configured to recognize tilt of an image-capturing direction of the image-capturing unit relative to an image-capturing direction in which the image-capturing unit being directly opposite to the subject captures the subject; a first correction unit configured to generate a corrected image by correcting distortion in the captured image due to the tilt; and a second correction unit configured to acquire a corrected captured image by correcting non-linear distortion in the corrected image.

TECHNICAL FIELD

The present invention relates to an image correction system, an imagecorrection method, and a computer program product.

BACKGROUND ART

It is known that there are two types of distortion, linear distortion(trapezoidal distortion) and non-linear distortion, that can occur in aprojected image projected by a projection apparatus such as a projector.Linear distortion occurs, for example, when the projection apparatusdeviates from a position directly opposite to a projection surface. Inother words, linear distortion occurs when a projection apparatusdesigned to be disposed in a position perpendicular to a projectionsurface is not directly opposite to the projection surface.

Non-linear distortion occurs, for example, when a projection surfacesuch as a hanging projector screen is uneven. In general, an unevenprojection surface such as a hanging projector screen lacks linearity,thus non-linear distortion occurs.

A projection system (projector and camera system) that includes aprojection apparatus and an image-capturing apparatus is a well-knowntechnology to correct distortion in a projected image. For example, thistechnology corrects distortion in a projected image projected by theprojection apparatus such as a projector on the basis of a capturedimage captured by a portable image-capturing apparatus (such as adigital camera, a web camera, and a camera built in a mobile phone or asmart phone, and hereinafter referred to as an “external camera”).

Patent document 1 is known as a technical document that discloses atechnique to correct distortion in a projected image. Patent document 1discloses an invention to correct distortion in a projected image byusing a captured image captured by a mobile phone with a built-incamera.

However, linear distortion occurs in a corrected projected image in somecases when a captured image acquired by an image-capturing apparatuscapturing a projected image is used, for example, in order to correctdistortion in the projected image due to unevenness of a projectionsurface.

CITATION LIST Patent Literature

[Patent Literature 1] Japanese Patent Application Laid-open No.2006-033357

DISCLOSURE OF INVENTION Problem to be Solved by the Invention

The present invention has been made in view of the disadvantagedescribed above, and it is an object of the present invention to providean image correction system, an image correction method and a computerprogram product with which, if a captured image acquired by animage-capturing apparatus capturing a subject is distorted, distortionof the subject contained in the captured image can be eliminated byutilizing the captured image.

Means for Solving Problem

To achieve the object, an image correction system according to thepresent invention includes: an image-capturing unit configured toacquire a captured image by capturing a subject; a tilt recognition unitconfigured to recognize tilt of an image-capturing direction of theimage-capturing unit relative to an image-capturing direction in whichthe image-capturing unit being directly opposite to the subject capturesthe subject; a first correction unit configured to generate a correctedimage by correcting distortion in the captured image due to the tilt;and a second correction unit configured to acquire a corrected capturedimage by correcting non-linear distortion in the corrected image.

An image correction method according to the present inventioncomprising: acquiring, by an image-capturing unit, a captured image bycapturing a subject; recognizing, by a tilt recognition unit, tilt of animage-capturing direction of the image-capturing unit relative to animage-capturing direction in which the image-capturing unit beingdirectly opposite to the subject captures the subject; generating, by afirst correction unit, a corrected image by correcting distortion in thecaptured image due to the tilt; and acquiring, by a second correctionunit, a corrected captured image by correcting non-linear distortion inthe corrected image.

A computer program product according to the present invention comprisinga non-transitory computer-usable medium having a computer program thatcauses a computer to function as: an image-capturing unit configured toacquire a captured image by capturing a subject; a tilt recognition unitconfigured to recognize tilt of an image-capturing direction of theimage-capturing unit relative to an image-capturing direction in whichthe image-capturing unit being directly opposite to the subject capturesthe subject; a first correction unit configured to generate a correctedimage by correcting distortion in the captured image due to the tilt;and a second correction unit configured to acquire a corrected capturedimage by correcting non-linear distortion in the corrected image.

Effect of the Invention

According to the present invention, if a captured image acquired by animage-capturing apparatus capturing a subject is distorted, thedistortion of the subject contained in the captured image can beeliminated by utilizing the captured image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example of a configuration offunctional blocks in an image correction system according to a firstembodiment;

FIG. 2 is a diagram illustrating an example of an embodiment of theimage correction system according to the first embodiment;

FIG. 3 is a diagram illustrating an example of a pattern image projectedby a projection unit of the image correction system according to thefirst embodiment;

FIG. 4 is a diagram illustrating an example of non-linear distortion ina projected image projected by the projection unit of the imagecorrection system according to the first embodiment;

FIG. 5 is a diagram illustrating an example of a case in which lineardistortion and non-linear distortion are contained in a captured imagecaptured by an image-capturing unit of the image correction systemaccording to the first embodiment;

FIG. 6 is a diagram illustrating an example of a projected image aftercorrection projected by the projection unit of the image correctionsystem according to the first embodiment;

FIG. 7 is a diagram illustrating an example of functional blocks in afirst correction unit of the image correction system according to thefirst embodiment;

FIG. 8 is a diagram illustrating an example of tilt, relative to aprojection surface, of the image-capturing unit of the image correctionsystem according to the first embodiment;

FIG. 9 is a diagram for explaining an example of a calculation method ofa projection transformation matrix calculated by a projectiontransformation matrix calculation unit of the image correction systemaccording to the first embodiment;

FIG. 10 is a flowchart for explaining an example of a generation methodof a corrected image by the image correction system according to thefirst embodiment;

FIG. 11 is a diagram illustrating an example of functional blocks in asecond correction unit of the image correction system according to thefirst embodiment;

FIG. 12 is a diagram for explaining an example of a correction methodused in the second correction unit of the image correction systemaccording to the first embodiment;

FIG. 13 is a diagram illustrating an example of relations amongprojection transformation matrices that correspond to projection,image-capturing, and correction by the image correction system accordingto the first embodiment;

FIG. 14 is a flowchart for explaining an example of a correction methodused in the second correction unit of the image correction systemaccording to the first embodiment;

FIG. 15 is a diagram for explaining an operation of the image correctionsystem according to the first embodiment;

FIG. 16 is a diagram illustrating a first modification of the patternimage projected by the projection unit of the image correction systemaccording to the first embodiment;

FIG. 17 is a diagram illustrating a second modification of the patternimage projected by the projection unit of the image correction systemaccording to the first embodiment;

FIG. 18 is a block diagram illustrating an example of a configuration offunctional blocks in an image correction system according to a secondembodiment;

FIG. 19 is a diagram illustrating an example of a subject of animage-capturing apparatus of the image correction system according tothe second embodiment;

FIG. 20 is a block diagram illustrating an example of a configuration offunctional blocks in an image correction system according to a thirdembodiment;

FIG. 21 a diagram illustrating an example of a subject of animage-capturing apparatus of the image correction system according tothe third embodiment;

FIG. 22 is a diagram illustrating an example of a subject of theimage-capturing apparatus of the image correction system according tothe third embodiment; and

FIG. 23 is a diagram for explaining an example of a hardwareconfiguration to execute software (computer program) in the imagecorrection systems according to the first to the third embodiments.

BEST MODES FOR CARRYING OUT THE INVENTION

Embodiments will now be described in detail below of an image correctionsystem, an image correction method and a computer program product withreference to the accompanying drawings.

First Embodiment

FIG. 1 is a block diagram illustrating an example of a configuration offunctional blocks in an image correction system 1 according to a firstembodiment. The image correction system 1 of the first embodimentincludes an image-capturing apparatus 2 and a projection apparatus 3.The image-capturing apparatus 2 includes an image-capturing unit 21 anda tilt recognition unit 22. The projection apparatus 3 includes acontroller 31, a projection unit 32, a first correction unit 33, and asecond correction unit 34.

The image-capturing unit 21 captures a projected image projected by theprojection unit 32. The image-capturing unit 21 transmits the capturedimage to the first correction unit 33.

The tilt recognition unit 22 recognizes tilt of an image-capturingdirection of the image-capturing unit 21 relative to an image-capturingdirection in which the image-capturing unit 21 being directly oppositeto a projection surface captures the projection surface. The tiltrecognition unit 22 recognizes the tilt with an accelerometer when theimage-capturing unit 21 captures a projected image. The accelerometer,for example, provides such information as the image-capturing unit 21 istilted by 10 degrees relative to the ground. The tilt recognition unit22 transmits information indicating the tilt (hereinafter referred to as“tilt information”) to the first correction unit 33.

The accelerometer will be described. When the accelerometer is athree-axis accelerometer, trapezoidal distortion (linear distortion) inthe longitudinal direction in a captured image can be corrected. Whenthe accelerometer is a four-axis to six-axis accelerometer, trapezoidaldistortion (linear distortion) in the lateral direction can also becorrected, and correction of gyro-like spin is also possible.

In general, a projection surface such as a screen on which theprojection unit 32 projects an image is perpendicular to the ground.This can lead to an assumption that, when an external camera(image-capturing apparatus 2) is directly opposite to the projectionsurface, the external camera (image-capturing apparatus 2) isperpendicular to the ground. Therefore, a corrected image acquired bycorrecting tilt of a captured image by using tilt information recognizedat the tilt recognition unit 22 is identical to a captured image (acaptured image captured from the front) that is acquired when theexternal camera being directly opposite to the projection surfacecaptures the projection surface such as a screen.

The image-capturing apparatus 2 may transmit a captured image and tiltinformation to the projection apparatus 3 in either of a wireless orwired manner.

The controller 31 inputs, into the projection unit 32, an input image tobe projected by the projection unit 32 as a projected image. Thecontroller 31 also inputs the input image into the second correctionunit 34 in order to correct non-linear distortion in the input image.The controller 31 receives, from the second correction unit 34, acorrected input image that is generated by correcting the non-lineardistortion in the input image. The controller 31 then inputs thecorrected input image into the projection unit 32.

The projection unit 32 projects the input image or the corrected inputimage as a projected image on a projection surface. The projectionsurface is, for example, a screen, a wall or a white board. The inputimage and the corrected input image will be described later in detail.

The first correction unit 33 generates a corrected image from a capturedimage on the basis of tilt information. The first correction unit 33transmits the corrected image to the second correction unit 34. Thefirst correction unit 33 will be described later in detail.

The second correction unit 34 corrects non-linear distortion in theinput image on the basis of the corrected image, and generates acorrected input image. The second correction unit 34 will be describedlater in detail.

FIG. 2 is a diagram illustrating an example of an embodiment of theimage correction system 1 according to the first embodiment. The imagecorrection system 1 in FIG. 2 is implemented by an external camera(image-capturing apparatus 2) and a short-throw projector (projectionapparatus 3). The short-throw projector (projection apparatus 3) isprojecting a pattern image for correcting distortion in a projectedimage on a screen (projection surface). Intersections of the linescontained in the pattern image in FIG. 2 are used by the secondcorrection unit 34, as corresponding points for associating a correctedimage generated by the first correction unit 33 with an input image.

FIG. 3 is a diagram illustrating an example of a pattern image projectedby the projection unit 32 of the image correction system 1 according tothe first embodiment. FIG. 3 is an example of a pattern image with agrid pattern.

In the example of FIG. 2, the short-throw projector (projectionapparatus 3) is projecting the pattern image. In the example of FIG. 2,non-linear distortion occurs in the projected pattern image because ofthe distortion of the screen (projection surface). The second correctionunit 34 corrects the non-linear distortion in the projected image byusing the intersections of the lines contained in the grid pattern imageas the corresponding points between a corrected image and an inputimage.

Described are an input image input by the controller 31 into theprojection unit 32, and distortion in a projected image. FIG. 4 is adiagram illustrating an example of non-linear distortion in a projectedimage projected by the projection unit 32 of the image correction system1 according to the first embodiment. The example of FIG. 4 illustrates acase in which a square is projected on the projection surface. In theexample of FIG. 4, non-linear distortion occurs in the square of theprojected image because of the distortion of the projection surface.

When the image-capturing unit 21 acquires a captured image to detectthis non-linear distortion, there is a case in which linear distortionoccurs in the captured image because the image-capturing unit 21deviates from a position directly opposite to the projection surface.

FIG. 5 is a diagram illustrating an example of a case in which lineardistortion and non-linear distortion are contained in a captured imagecaptured by the image-capturing unit 21 of the image correction system 1according to the first embodiment. The example of the captured image inFIG. 5 is acquired, for example, when the external camera(image-capturing apparatus 2) rotates horizontally to the ground in theright direction about the y-axis of the coordinate axes of the externalcamera (image-capturing apparatus 2) in FIG. 2, and deviates from aposition directly opposite to the screen (projection surface).

The first correction unit 33 corrects a captured image containing lineardistortion and non-linear distortion as illustrated in FIG. 5 byeliminating the linear distortion due to deviation of theimage-capturing unit 21 from a position directly opposite to theprojection surface, and generates a corrected image from the capturedimage.

FIG. 6 is a diagram illustrating an example of a projected image aftercorrection projected by the projection unit 32 of the image correctionsystem 1 according to the first embodiment. The second correction unit34 generates a corrected input image on the basis of the input image andthe corrected image generated by the first correction unit 33 from thecaptured image. When the projection unit 32 projects the corrected inputimage, a projected image after correction is displayed on the projectionsurface. The projected image after correction is identical to the inputimage.

Next, described in detail is the first correction unit 33 in the imagecorrection system 1 according to the first embodiment. FIG. 7 is adiagram illustrating an example of functional blocks in the firstcorrection unit 33 of the image correction system 1 of the firstembodiment. The first correction unit 33 includes an intrinsic parametermatrix calculation unit 331, a rotation matrix calculation unit 332, athree-dimensional coordinate calculation unit 333, a projectiontransformation matrix calculation unit 334, and a corrected imagegeneration unit 335.

The intrinsic parameter matrix calculation unit 331 calculates anintrinsic parameter matrix of the image-capturing unit 21. First, theintrinsic parameter matrix is described.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack} & \; \\{A = \begin{pmatrix}f & 0 & {cx} \\0 & f & {cy} \\0 & 0 & 1\end{pmatrix}} & (1)\end{matrix}$

Formula 1 is an intrinsic parameter matrix A. Described are componentsof the intrinsic parameter matrix A of Formula 1, where f denotes thefocal length of the image-capturing unit 21, cx denotes x-coordinate ofthe principal point (cx, cy) of the image-capturing unit 21, and cydenotes y-coordinate of the principal point (cx, cy) of theimage-capturing unit 21.

The intrinsic parameter matrix calculation unit 331 needs to acquire thefocal length f and the principal point (cx, cy) of the image-capturingunit 21 in order to calculate the intrinsic parameter matrix A.

The intrinsic parameter matrix calculation unit 331 may acquire thefocal length f and the principal point (cx, cy) on the basis ofexchangeable image file format (Exif) information that is taginformation of a captured image. The intrinsic parameter matrixcalculation unit 331 may acquire the focal length f and the principalpoint (cx, cy) by camera calibration of the image-capturing unit 21.

The intrinsic parameter matrix A is used when the three-dimensionalcoordinate calculation unit 333 calculates, from two-dimensionalcoordinates of the captured image, three-dimensional coordinates in thereal space corresponding to the two-dimensional coordinates.

The rotation matrix calculation unit 332 acquires tilt information fromthe tilt recognition unit 22. The tilt information includes informationindicating a tilt θ of an image-capturing direction of theimage-capturing unit 21 relative to an image-capturing direction inwhich the image-capturing unit 21 being directly opposite to theprojection surface captures the projection surface. The rotation matrixcalculation unit 332 calculates a rotation matrix R representingcoordinate transformation that rotates coordinates by just the tilt θ.

The rotation matrix R will now be described. FIG. 8 is a diagramillustrating an example of the tilt θ, relative to the projectionsurface, of the image-capturing unit 21 of the image correction system 1according to the first embodiment. The example of FIG. 8 illustrates acase in which an image-capturing direction (z-axis direction) of theimage-capturing apparatus 2 is rotated by just the tilt θ around thex-axis so that the image-capturing unit 21 is directly opposite to theprojection surface. In FIG. 8, x, y₁, and z₁ denote the coordinate axesbefore rotation by just the tilt θ, and x, y₂, and z₂ denote thecoordinate axes after rotation by just the tilt θ.

Coordinates (x, y₂, z₂) can be calculated from the following formula byusing the rotation matrix R that represents rotation by just the tilt θ.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack} & \; \\{\begin{pmatrix}x \\y_{2} \\z_{2}\end{pmatrix} = {\begin{pmatrix}1 & 0 & 0 \\0 & {\cos\;\theta} & {{- \sin}\;\theta} \\0 & {\sin\;\theta} & {\cos\;\theta}\end{pmatrix}\begin{pmatrix}x \\y_{1} \\z_{1}\end{pmatrix}}} & (2)\end{matrix}$

The rotation matrix calculation unit 332 calculates a matrix thatsatisfies Formula 2 as the rotation matrix R.

The three-dimensional coordinate calculation unit 333 calculatesthree-dimensional coordinates corresponding to coordinates of a capturedimage by perspective projection transformation in which the coordinatesof the captured image and the intrinsic parameter matrix A of theimage-capturing unit 21 are used. Perspective projection transformationwill now be described. Capturing a projected image by theimage-capturing unit 21 corresponds to an operation of transforming thethree-dimensional coordinates in the real space into the two-dimensionalcoordinates in the captured image. This transformation is calledperspective projection transformation. Perspective projectiontransformation can be represented by a perspective projectiontransformation matrix P of the image-capturing unit 21. The perspectiveprojection transformation matrix P is expressed by the followingformula.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 3} \right\rbrack} & \; \\{P = {{A\lbrack{Rt}\rbrack} = {\begin{pmatrix}f & 0 & {cx} \\0 & f & {cy} \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}r_{11} & r_{12} & r_{13} & t_{1} \\r_{21} & r_{22} & r_{23} & t_{2} \\r_{31} & r_{32} & r_{33} & t_{3}\end{pmatrix}}}} & (3)\end{matrix}$

The matrix A in Formula 3 is the above-mentioned intrinsic parametermatrix of the image-capturing unit 21. A matrix [Rt] is an extrinsicparameter matrix. The extrinsic parameter matrix represents rotation andtranslation of the image-capturing unit 21.

When the image-capturing unit 21 is tilted by just the tilt θ asillustrated in FIG. 8 and captures a projected image, a perspectiveprojection transformation matrix P₁ is expressed by the followingformula.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 4} \right\rbrack} & \; \\{P_{1} = {\begin{pmatrix}f & 0 & {cx} \\0 & f & {cy} \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}1 & 0 & 0 & 0 \\0 & {\cos\;\theta} & {{- \sin}\;\theta} & 0 \\0 & {\sin\;\theta} & {\cos\;\theta} & 0\end{pmatrix}}} & (4)\end{matrix}$

Formula 4 defines t=0 because the image-capturing unit 21 itself is theorigin, and because the image-capturing unit 21 rotates about theorigin.

The relation between the three-dimensional coordinates in the real spaceand the two-dimensional coordinates in the captured image can berepresented as follows.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 5} \right\rbrack} & \; \\{\begin{pmatrix}x_{p} \\y_{p} \\1\end{pmatrix} \cong {\begin{pmatrix}f & 0 & {cx} \\0 & f & {cy} \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}1 & 0 & 0 & 0 \\0 & {\cos\;\theta} & {{- \sin}\;\theta} & 0 \\0 & {\sin\;\theta} & {\cos\;\theta} & 0\end{pmatrix}\begin{pmatrix}X \\Y \\Z \\1\end{pmatrix}}} & (5)\end{matrix}$

Formula 5 represents equality except for scale. That is, Formula 5represents equality in a homogeneous coordinate system.

When the image-capturing unit 21 being directly opposite to theprojection surface captures the projection surface, a perspectiveprojection transformation matrix P₂ is expressed by the followingformula.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 6} \right\rbrack} & \; \\{P_{2} = {\begin{pmatrix}f & 0 & {cx} \\0 & f & {cy} \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & 0\end{pmatrix}}} & (6)\end{matrix}$

The relation between the three-dimensional coordinates in the real spaceand the two-dimensional coordinates in the captured image can berepresented as follows.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 7} \right\rbrack} & \; \\{\begin{pmatrix}x_{1} \\y_{1} \\1\end{pmatrix} \cong {\begin{pmatrix}f & 0 & {cx} \\0 & f & {cy} \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & 0\end{pmatrix}\begin{pmatrix}X \\Y \\Z \\1\end{pmatrix}}} & (7)\end{matrix}$

In general, Formula 7 is equivalent to Formula 8 below. In Formula 7, Ris defined as the identity matrix, and t is defined as 0. Thus, Formula8 below defines R=E (identity matrix) and t=0.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 8} \right\rbrack} & \; \\\left\{ \begin{matrix}{\begin{pmatrix}x \\y \\z\end{pmatrix} = {{R\begin{pmatrix}X \\Y \\Z\end{pmatrix}} + t}} \\{x^{\prime} = {x/z}} \\{y^{\prime} = {y/z}} \\{x_{1} = {{f*x^{\prime}} + {cx}}} \\{y_{1} = {{f*y^{\prime}} + {cy}}}\end{matrix} \right. & (8)\end{matrix}$

The three-dimensional coordinate calculation unit 333 calculatesthree-dimensional coordinates in the real space corresponding to thecoordinates of the captured image by using an inverse transformationmatrix P₁ ⁻¹ that is the inverse of the perspective projectiontransformation matrix P₁. The three-dimensional coordinate calculationunit 333 does not need to calculate the corresponding three-dimensionalcoordinates in the real space with respect to all the points in thetwo-dimensional coordinates of the captured image. In other words, thethree-dimensional coordinate calculation unit 333 may calculatecorresponding points in the three-dimensional coordinates correspondingto the points in the captured image as many a number as needed (four) tocalculate a projection transformation matrix H to be described later.

The projection transformation matrix calculation unit 334 calculates theprojection transformation matrix H that transforms the coordinates ofthe captured image into coordinates of the corrected image. Theprojection transformation matrix H is used by the corrected imagegeneration unit 335 to calculate the coordinates of the corrected imagecorresponding to the coordinates of the captured image.

FIG. 9 is a diagram for explaining an example of a calculation method ofthe projection transformation matrix H calculated by the projectiontransformation matrix calculation unit 334 of the image correctionsystem 1 according to the first embodiment. The projectiontransformation matrix H is expressed by the following formula.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 9} \right\rbrack} & \; \\{p_{2} \cong {Hp}_{1}} & (9) \\{\left\lbrack {{Math}.\mspace{14mu} 10} \right\rbrack\mspace{616mu}} & \; \\{\begin{pmatrix}x_{2} \\y_{2} \\1\end{pmatrix} \cong {\begin{pmatrix}h_{1} & h_{2} & h_{3} \\h_{4} & h_{5} & h_{6} \\h_{7} & h_{8} & 1\end{pmatrix}\begin{pmatrix}x_{1} \\y_{1} \\1\end{pmatrix}}} & (10)\end{matrix}$

Formula 9 and Formula 10 represent equality except for scale. That is,Formula 9 and Formula 10 represent equality in a homogeneous coordinatesystem.

Calculating coefficients of the projection transformation matrix Hresults in calculating Formula 11 below.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 11} \right\rbrack\mspace{616mu}} & \; \\\left\{ \begin{matrix}{x_{2} = \frac{{h_{1}x_{1}} + {h_{2}y_{1}} + h_{3}}{{h_{7}x_{1}} + {h_{8}y_{1}} + 1}} \\{y_{2} = \frac{{h_{4}x_{1}} + {h_{5}y_{1}} + h_{6}}{{h_{7}x_{1}} + {h_{8}y_{1}} + 1}}\end{matrix} \right. & (11)\end{matrix}$

Formula 11 includes eight unknown coefficients h₁ to h₈. Formula 11 isobtained for each pair of coordinates (x₁, y₁) of the captured image andcoordinates (x₂, y₂) of the corrected image corresponding to thecoordinates (x₁, y₁) of the captured image. In order to determine theeight unknown coefficients h₁ to h₈, information on at least four pointsof the corresponding points in the corrected image corresponding to thepoints in the captured image. The coordinates (x₂, y₂) of the correctedimage corresponding to the coordinates (x₁, y₁) of the captured imagecan be calculated by applying the perspective projection transformationmatrix P₂ to the three-dimensional coordinates in the real spacecorresponding to the above-described coordinates of the captured image.

The projection transformation matrix calculation unit 334 may calculatethe projection transformation matrix H by the least-square method thatuses a plurality of pieces of information on the corresponding points inthe corrected image corresponding to the points in the captured image.Thereby, if the above-described corresponding points acquired by theperspective projection transformation matrix calculation unit 334contain some errors with respect to correspondence, a projectiontransformation matrix H can be calculated that contains minimum errors.The least-square method is not the only method for reducing the effectof errors with respect to correspondence, and other methods can be usedfor this purpose.

The corrected image generation unit 335 generates a corrected image byapplying the projection transformation matrix H to all the pixelcoordinates in the captured image.

FIG. 10 is a flowchart for explaining an example of a generation methodof the corrected image by the image correction system 1 according to thefirst embodiment.

The first correction unit 33 receives a captured image from theimage-capturing unit 21. The first correction unit 33 also receives tiltinformation from the tilt recognition unit 22 (Step S1). The intrinsicparameter matrix calculation unit 331 calculates an intrinsic parametermatrix A of the image-capturing unit 21 on the basis of Exif informationthat is tag information of the captured image (Step S2). The rotationmatrix calculation unit 332 calculates a rotation matrix R on the basisof the tilt information (Step S3).

The three-dimensional coordinate calculation unit 333 calculates aperspective projection transformation matrix P₁ from the intrinsicparameter matrix A of the image-capturing unit 21 and the rotationmatrix R. The three-dimensional coordinate calculation unit 333calculates three-dimensional coordinates corresponding to coordinates ofthe captured image by applying an inverse transformation matrix P₁ ⁻¹that is the inverse of the perspective projection transformation matrixP₁ to the coordinates of the captured image (Step S4). The projectiontransformation matrix calculation unit 334 calculates coordinates of acorrected image corresponding to the three-dimensional coordinates byapplying a perspective projection transformation matrix P₂ to thethree-dimensional coordinates (Step S5).

At Step S4 and Step S5, the first correction unit 33 calculatescorresponding points of the coordinates in the corrected imagecorresponding to points of the coordinates in the captured image. Theprojection transformation matrix calculation unit 334 calculates aprojection transformation matrix H by the least-square method, in whichthe corresponding points of the coordinates in the corrected imagecorresponding to the points of the coordinates in the captured image isused (Step S6). The corrected image generation unit 335 generates thecorrected image by applying the projection transformation matrix H toall the pixel coordinates in the captured image (Step S7).

Next, described is the second correction unit 34 in the image correctionsystem 1 according to the first embodiment. FIG. 11 is a diagramillustrating an example of functional blocks in the second correctionunit 34 of the image correction system 1 of the first embodiment. Thesecond correction unit 34 includes a correction region determinationunit 341, a corresponding point detection unit 342, a projectiontransformation matrix calculation unit 343, and a corrected input imagegeneration unit 344.

The correction region determination unit 341 determines a correctionregion in the corrected image. FIG. 12 is a diagram for explaining anexample of a correction method used in the second correction unit 34 ofthe image correction system 1 according to the first embodiment. Aregion 41 in FIG. 12 is an example of the correction region determinedby the correction region determination unit 341. The correction regiondetermination unit 341 may determine the region 41 by any method. Forexample, the correction region determination unit 341 determines, as thecorrection region, a rectangle having the equal aspect ratio to that ofthe projection region of the projected image.

Pixel values of an input image are associated with pixel values of thecorrection region. In general, a correction region is smaller than aninput image in size. Thus, the input image needs to be reduced so thatthe input image is fully displayed on the correction region.

Back to FIG. 11, the corresponding point detection unit 342 detectscorresponding points in the pattern image on a corrected input imagecorresponding to points in a pattern image on the corrected image. Forexample, in the example of FIG. 12, the corresponding point detectionunit 342 associates a point 43 in the corrected image with a point 53 inthe corrected input image. The corresponding point detection unit 342associates a point 44 in the corrected image with a point 54 in thecorrected input image. The corresponding point detection unit 342associates a point 45 in the corrected image with a point 55 in thecorrected input image. The corresponding point detection unit 342associates a point 46 in the corrected image with a point 56 in thecorrected input image. Thereby, a unit region 42 in the corrected imageis associated with a unit region 52 in the corrected input image.

The projection transformation matrix calculation unit 343 determines aprojection transformation matrix H_(n)′ for each unit region n. Forexample, the projection transformation matrix calculation unit 343determines the projection transformation matrix H_(n)′ on the basis ofFormula 11 obtained by four points detected by the corresponding pointdetection unit 342.

The corrected input image generation unit 344 determines pixel values ofcoordinates obtained by applying the projection transformation matrixH_(n)′ to coordinates in the correction region on the basis of pixelvalues of the coordinates in the correction region to which theprojection transformation matrix H_(n)′ is applied. The pixel values arenot based on pixel values of the correction region in the correctedimage but on pixel values of the input image fitted to the region 41.

For example, the corrected input image generation unit 344 determines apixel value 57 of the corrected input image on the basis of a pixelvalue 47 of the input image in a position corresponding to a position inthe correction region associated by the projection transformation matrixH₁′. The corrected input image generation unit 344 determines a pixelvalue of a certain point in the corrected input image by the projectiontransformation matrix H_(n)′. Thereby, the corrected input imagegeneration unit 344 generates a corrected input image 51.

FIG. 13 is a diagram illustrating an example of relations amongprojection transformation matrices that correspond to projection,image-capturing, and correction by the image correction system 1according to the first embodiment. Projecting an input image by theprojection unit 32 corresponds to the transformation represented by aninverse matrix (H_(n)′)⁻² that is the inverse of the projectiontransformation matrix H_(n)′. Capturing a projected image corresponds tothe transformation represented by the inverse matrix H⁻¹ that is theinverse of the projection transformation matrix H calculated by thefirst correction unit 33. Therefore, the projected image and thecorrected image are identical. That is, a matrix E is the identitymatrix.

The second correction unit 34 transforms coordinates of the correctionregion in the corrected image into coordinates of the corrected inputimage by the projection transformation matrix H_(n)′. The secondcorrection unit 34 determines pixel values of the coordinates in thecorrected input image to be the same pixel values as those in the inputimage that has been fitted to the correction region. Thus, an image inthe correction region and a projected image after correction areidentical. That is, a matrix E_(n) is the identity matrix.

FIG. 14 is a flowchart for explaining an example of a correction methodused in the second correction unit 34 of the image correction system 1according to the first embodiment.

The corresponding point detection unit 342 calculates correspondingpoints between a corrected image and a corrected input image by usinggrid points in the grid pattern image (Step S11). This associates unitregions in the corrected image with unit regions in the corrected inputimage. The correction region determination unit 341 determines acorrection region to be corrected by the second correction unit 34 (StepS12). The correction region determination unit 341 associates pixelvalues of the input image with pixel values of the correction region(Step S13). The projection transformation matrix calculation unit 343calculates projection transformation matrices for respective unitregions in the correction region (Step S14). The corrected input imagegeneration unit 344 transforms certain coordinates in the correctionregion in the corrected image into coordinates of the corrected inputimage by the projection transformation matrix H_(n)′. The correctedinput image generation unit 344 generates the corrected input image bydetermining pixel values of the coordinates in the corrected input imageto be the same pixel values as those in the input image that has beenfitted to the correction region (Step S15).

FIG. 15 is a diagram for explaining an operation of the image correctionsystem 1 according to the first embodiment.

The controller 31 transmits to the projection unit 32 a signalindicating the start of calibration of a projected image (Step S21). Theprojection unit 32 projects a pattern image as the projected image on aprojection surface (Step S22). The image-capturing unit 21 captures theprojected image and acquires a captured image (Step S23). The tiltrecognition unit 22 recognizes a tilt θ of the image-capturing unit 21at the time of capturing at Step S23 (Step S24). The first correctionunit 33 generates a corrected image by correcting the captured image onthe basis of an intrinsic parameter matrix of the image-capturing unit21 and the tilt θ recognized at the time of capturing (Step S25).

The second correction unit 34 calculates, as distortion correctionparameters, a correction region in the corrected image, and a projectiontransformation matrix calculated for each unit region and used fortransforming a corrected image into a corrected input image (Step S26).The following describes S26 in detail. First, the correction region inthe corrected image is described. The correction region in the correctedimage will be a projection region in a corrected projected image. Thus,the second correction unit 34 determines an aspect ratio of thecorrection region to be equal to that of an input image. In general, acorrected projected image (corrected input image) projected on aprojection surface corresponding to a correction region in a correctedimage is a reduced input image.

Next, described is the projection transformation matrix calculated foreach unit region and used for transforming a corrected image into acorrected input image. The second correction unit 34 detectscorresponding points between a pattern image on the corrected image andthat on the input image. The second correction unit 34 calculates aprojection transformation matrix for each unit region defined by linesegments joining the corresponding points.

The second correction unit 34 stores the distortion correction parameterinto a memory of the image correction system 1 (Step S27). The secondcorrection unit 34 receives an input image from the controller 31 (StepS28). The second correction unit 34 reads out the distortion correctionparameter that has been stored at Step S27 from the memory (Step S29).The second correction unit 34 corrects the input image on the basis ofthe distortion correction parameter (Step S30).

The controller 31 determines whether or not it has received a commandthat instructs termination of calibration of the projected image (StepS31). When the controller 31 receives a command that instructstermination of calibration (Yes at Step S31), the controller 31terminates the process. When the controller 31 does not receive acommand that instructs termination of calibration (No at Step S31), theprocess returns to Step S28.

The pattern image used in the image correction system 1 according to thefirst embodiment may be a pattern image other than the grid patternimage. FIG. 16 is a diagram illustrating a first modification of thepattern image projected, by the projection unit 32 of the imagecorrection system 1 of the first embodiment. The example of FIG. 16illustrates a pattern image of a chessboard pattern. FIG. 17 is adiagram illustrating a second modification of the pattern imageprojected by the projection unit 32 of the image correction system 1 ofthe first embodiment. The example of FIG. 17 illustrates a pattern imagewith circles.

The corresponding point detection unit 342 of the second correction unit34 may detect corresponding points without using a pattern image. Forexample, the corresponding point detection unit 342 may apply, to acontent image of a user, a corresponding-point extraction method inwhich scale invariant feature transform (SIFT) is used. When thecontroller 31 determines that the content image of a user has few imagefeatures, the controller 31 may determine not to apply thecorresponding-point extraction method by SIFT to the content image.Specifically, the controller 31 may calculate frequencies of contentimages, and determine not to use a content image with few high-frequencycomponents.

The correction processing by the first correction unit 33 and the secondcorrection unit 34 may be performed in an external computer connectedthrough a network instead of in the projection apparatus 3. The imagecorrection system 1 may be connected with the external computer ineither of a wired or wireless manner.

Using an accelerometer is not the only method for the tilt recognitionunit 22 to recognize tilt of the image-capturing apparatus 2. Forexample, the tilt recognition unit 22 may recognize tilt of theimage-capturing apparatus 2 on the basis of image processing in which acaptured image captured by the image-capturing apparatus 2 is used. Inthe image processing, for example, the tilt recognition unit 22 mayrecognize the tilt of the image-capturing apparatus 2 by detecting anupper and a lower black edges of a projection surface (such as a screen)contained in a captured image, and using the ratio of the length of theupper black edge to that of the lower black edge.

Second Embodiment

Next, described is an image correction system 1 according to a secondembodiment. FIG. 18 is a block diagram illustrating an example of aconfiguration of functional blocks in the image correction system 1 ofthe second embodiment. The image correction system 1 of the secondembodiment differs from the image correction system 1 of the firstembodiment in that the image-capturing apparatus 2 includes the firstcorrection unit 33 and the second correction unit 34. This allows theprojection apparatus 3 to have a simple configuration that includes onlybasic functions for projection. The image correcting system 1 of thesecond embodiment also differs in that it includes a storage unit 23that stores a corrected captured image acquired by correcting non-lineardistortion in a corrected image by the second correction unit 34.Detailed description of the operation of the image correction system 1according to the second embodiment is omitted because the descriptionthereof is the same as that of the operation of the image correctionsystem 1 according to the first embodiment.

In the description of the image correction system 1 of the firstembodiment, the projection apparatus 3 projects an image on a screen asa projection surface, and the image-capturing apparatus 2 captures theprojection surface, for example. However, the projection surface for theprojection apparatus 3 and the subject for the image-capturing apparatus2 are not limited to a screen. In the description of the imagecorrection system 1 of the second embodiment, described is a case inwhich the subject of the image-capturing apparatus 2 is not the screen.

FIG. 19 is a diagram illustrating an example of a subject of theimage-capturing apparatus 2 of the image correction system 1 accordingto the second embodiment. The example of FIG. 19 illustrates a case inwhich the projection apparatus 3 projects the pattern image with circlesillustrated in FIG. 17 on a book, and the image-capturing apparatus 2acquires a captured image by capturing the book. Following descriptionis a case of correcting a captured image acquired by capturing a book.First, the image-capturing unit 21 acquires a captured image bycapturing a book on which a pattern image is projected. Then, the firstcorrection unit 33 acquires a corrected image by correcting trapezoidaldistortion (linear distortion) in the captured image because of theimage-capturing apparatus 2 not being directly opposite to the book asthe subject when capturing it.

Next, the second correction unit 34 acquires a corrected captured imageby correcting non-linear distortion caused by, for example, a bend inthe book. Specifically, the second correction unit 34 receives thepattern image illustrated in FIG. 17 as an input image, and calculatesthe above-described projection transformation matrix H_(n)′ from theinput image and the corrected image. In other words, the secondcorrection unit 34 calculates the above-described projectiontransformation matrix H_(n)′ by establishing correspondences between thecircles in the pattern image illustrated in FIG. 17 and the circles inthe pattern image contained in the corrected image. The pattern imageprojected by the projection unit 32 is not limited to the pattern imageillustrated in FIG. 17, but any pattern image can be used in order forthe second correction unit 34 to calculate the projection transformationmatrix H_(n)′. The second correction unit 34 stores projectiontransformation matrices H_(n)′ for respective regions n (regionsincluding respective circles in the pattern image) in the storage unit23. Thereby, the second correction unit 34 can correct non-lineardistortion identical to the non-linear distortion caused by, forexample, a bend in the book illustrated in FIG. 19. For example, theprojection transformation matrices H_(n)′ stored in the storage unit 23can be used in a case in which the subject is a book that has the sameshape as that of the book in FIG. 19 and on which the same non-lineardistortion occurs as that on the book in FIG. 19.

The second correction unit 34 acquires a corrected captured image bytransforming the respective regions n in the corrected image by theprojection transformation matrices H_(n)′, and stores the correctedcaptured image in the storage unit 23. The image-capturing apparatus 2may transmits the corrected captured image to the projection apparatus3, and then the projection apparatus 3 may project the correctedcaptured image. Thereby, when a book is captured as a subject to acquirea captured image, and then the first correction unit 33 and the secondcorrection unit 34 correct the captured image to acquire a correctedcaptured image, the corrected captured image can be displayed as aprojected image on a projection surface such as a screen.

Third Embodiment

Next, described is an image correction system 1 according to a thirdembodiment. FIG. 20 is a block diagram illustrating an example of aconfiguration of functional blocks in the image correction system 1 ofthe third embodiment. The image correction system 1 of the thirdembodiment includes the image-capturing apparatus 2 and a displayapparatus 4. The image correction system 1 of the third embodimentdiffers from the image correction systems 1 of the first and the secondembodiments in that it includes the display apparatus 4 instead of theprojection apparatus 3. The display apparatus 4 includes a display unit5 that displays a corrected captured image.

The configuration of the image-capturing apparatus 2 of the imagecorrection system 1 of the third embodiment is the same as that of thesecond embodiment, and thus the description thereof is omitted. Thedetailed description of the operation of the image correction system 1of the third embodiment is also omitted because the description thereofis the same as that of the image correction systems 1 of the first andthe second embodiments.

FIGS. 21 and 22 are diagrams each illustrating an example of a subjectof the image-capturing apparatus 2 of the image correction system 1according to the third embodiment. FIG. 21 is an example of a bookcontaining pages on which a pattern image is printed. FIG. 22 is anexample of a sheet of paper on which a pattern image is printed.Following description is an example of a case in which the subject ofthe image-capturing apparatus 2 is the book illustrated in FIG. 21. Itshould be noted that the following description is also applicable to acase in which the subject of the image-capturing apparatus 2 is thesheet of paper illustrated in FIG. 22.

First, the image-capturing unit 21 acquires a captured image bycapturing a book on which a pattern image is printed. Then, the firstcorrection unit 33 acquires a corrected image by correcting trapezoidaldistortion (linear distortion) in the captured image because of theimage-capturing apparatus 2 not being directly opposite to the book asthe subject when capturing it. The second correction unit 34 reads outthe same pattern image as the pattern image printed on the book from thestorage unit 23, and then calculates the above-described projectiontransformation matrix H_(n)′ from the pattern image and the correctedimage.

The second correction unit 34 stores, in the storage unit 23, projectiontransformation matrices H_(n)′ for respective regions n (regionsincluding respective circles in the pattern image) in the correctedimage. Thereby, the second correction unit 34 can correct non-lineardistortion that is identical to the non-linear distortion caused by, forexample, a bend in the book illustrated in FIG. 21. The secondcorrection unit 34 acquires a corrected captured image by transformingthe respective regions n in the corrected image by the projectiontransformation matrices, H_(n)′, and stores the corrected captured imagein the storage unit 23. The image-capturing apparatus 2 transmits thecorrected captured image to the display apparatus 4 to display thecorrected captured image on the display unit 5 of the display apparatus4. Thereby, when a book is captured as a subject to acquire a capturedimage, and the first correction unit 33 and the second correction unit34 corrects the captured image to acquire a corrected captured image,the corrected captured image can be displayed on the display unit 5 ofthe display apparatus 4.

The tilt recognition unit 22, the controller 31, the first correctionunit 33, and the second correction unit 34 of the image correctionsystems 1 according to the first to the third embodiments may beimplemented either by hardware such as an integrated circuit (IC), or bysoftware (computer program). The tilt recognition unit 22, thecontroller 31, the first correction unit 33, and the second correctionunit 34 of the image correction systems 1 of the first to the thirdembodiments may be implemented by combining hardware and software(computer program).

FIG. 23 is a diagram for explaining an example of a hardwareconfiguration to execute software (computer program) in the imagecorrection systems 1 according to the first to the third embodiments.The image-capturing apparatus 2 of the first to the third embodiments,and the projection apparatus 3 of the first and the second embodimentsinclude an auxiliary memory 61 such as a memory card, a main memory 62such as a random access memory (RAM), a processor 63 such as a centralprocessing unit (CPU), and a communication interface (I/F) 64 forcommunicating with other apparatuses. The auxiliary memory 61, the mainmemory 62, the processor 63, and the communication I/F 64 are connectedone another through a bus 65.

The auxiliary memory 61 stores software (computer program). However, thesoftware (computer program) may be provided as a computer programproduct recorded in a file in an installable or executable format in arecording medium, such as a CD-ROM, a flexible disk (FD), a CD-R, and adigital versatile disk (DVD), that is readable by a computer.

The software (computer program) may be stored on a computer connected toa network such as the Internet, and may be provided by downloadingthrough the network. The software (computer program) may also beprovided or distributed through a network such as the Internet.

The processor 63 reads out the software (computer program) stored in theauxiliary memory 61 such as a memory card, a storage medium such as aCD-ROM, or a computer on a network to execute it on the main memory 62.Thereby, when the tilt recognition unit 22, the controller 31, the firstcorrection unit 33, and the second correction unit 34 are implemented bysoftware (computer program), these functional blocks are implemented onthe main memory 62.

INDUSTRIAL APPLICABILITY

The image correction systems 1 according to the first to the thirdembodiments can provide an image correction system, an image correctionmethod and a computer program that can eliminate distortion in a subjectcontained in a captured image by using the captured image even if thecaptured image acquired by the image-capturing apparatus 2 capturing thesubject is distorted.

Although the invention has been described with respect to specificembodiments for a complete and clear disclosure, the appended claims arenot to be thus limited but are to be construed as embodying allmodifications and alternative constructions that may occur to oneskilled in the art that fairly fall within the basic teaching herein setforth.

EXPLANATION OF LETTERS OR NUMERALS

-   1 Image correction system-   2 Image-capturing apparatus-   3 Projection apparatus-   4 Display apparatus-   5 Display unit-   21 Image-capturing unit-   22 Tilt recognition unit-   23 Storage unit-   31 Controller-   32 Projection unit-   33 First correction unit-   34 Second correction unit-   331 Intrinsic parameter matrix calculation unit-   332 Rotation matrix calculation unit-   333 Three-dimensional coordinate calculation unit-   334 Projection transformation matrix calculation unit-   335 Corrected image generation unit-   341 Correction region determination unit-   342 Corresponding point detection unit-   343 Projection transformation matrix calculation unit-   344 Corrected input image generation unit-   61 Auxiliary memory-   62 Main memory-   63 Processor-   64 Communication I/F-   65 Bus-   A Intrinsic parameter matrix-   H Projection transformation matrix-   H₁′ Projection transformation matrix-   H_(n)′ Projection transformation matrix-   P Perspective projection transformation matrix-   P₁ Perspective projection transformation matrix-   P₂ Perspective projection transformation matrix-   R Rotation matrix-   f Focal length-   θ Tilt

The invention claimed is:
 1. An image correction system comprising: animage-capturing unit configured to acquire a captured image by capturinga subject; a tilt recognition unit configured to recognize tilt of animage-capturing direction of the image-capturing unit relative to animage-capturing direction in which the image-capturing unit beingdirectly opposite to the subject captures the subject; a firstcorrection unit configured to generate a corrected image by correctingdistortion in the captured image on the basis of information indicatingthe tilt; and a second correction unit configured to acquire a correctedcaptured image by correcting non-linear distortion in the correctedimage, wherein the tilt recognition unit recognizes the tilt with anaccelerometer.
 2. The image correction system according to claim 1,wherein the first correction unit comprises: a projection transformationmatrix calculation unit configured to calculate coordinates of thecorrected image corresponding to coordinates of the captured image, andcalculate a projection transformation matrix that transforms thecoordinates of the captured image into the coordinates of the correctedimage; and a corrected image generation unit configured to generate thecorrected image by applying the projection transformation matrix tocoordinates of all pixels in the captured image.
 3. The image correctionsystem according to claim 2, wherein the first correction unit furthercomprises: an intrinsic parameter matrix calculation unit configured tocalculate an intrinsic parameter matrix of the image-capturing unit; arotation matrix calculation unit configured to calculate a rotationmatrix representing coordinate transformation due to the tilt; and athree-dimensional coordinate calculation unit configured to calculatethree-dimensional coordinates corresponding to the coordinates of thecaptured image by perspective projection transformation in which thecoordinates of the captured image, the intrinsic parameter matrix of theimage-capturing unit, and the rotation matrix are used, and theprojection transformation matrix calculation unit calculates, from thethree-dimensional coordinates, the coordinates of the corrected imagecorresponding to the coordinates of the captured image by perspectiveprojection transformation in which the intrinsic parameter matrix isused.
 4. The image correction system according to claim 1, furthercomprising: a projection unit configured to project an image on aprojection surface, wherein the image-capturing unit acquires a capturedimage by capturing the projected image as the subject; the secondcorrection unit acquires a corrected projected image by correctingnon-linear distortion in the corrected image; and the projection unitfurther projects the corrected projected image on the projectionsurface.
 5. The image correction system according to claim 1, whereinthe tilt recognition unit recognizes the tilt when the image-capturingunit captures the subject.
 6. An image correction method comprising:acquiring, by an image-capturing unit, a captured image by capturing asubject; recognizing, by a tilt recognition unit, tilt of animage-capturing direction of the image-capturing unit relative to animage-capturing direction in which the image-capturing unit beingdirectly opposite to the subject captures the subject, the tiltrecognition unit recognizing the tilt with an accelerometer; generating,by a first correction unit, a corrected image by correcting distortionin the captured image on the basis of information indicating the tilt;and acquiring, by a second correction unit, a corrected captured imageby correcting non-linear distortion in the corrected image.
 7. Acomputer program product comprising a non-transitory computer-usablemedium having a computer program that causes a computer to function as:an image-capturing unit configured to acquire a captured image bycapturing a subject; a tilt recognition unit configured to recognizetilt of an image-capturing direction of the image-capturing unitrelative to an image-capturing direction in which the image-capturingunit being directly opposite to the subject captures the subject; afirst correction unit configured to generate a corrected image bycorrecting distortion in the captured image on the basis of informationindicating the tilt; and a second correction unit configured to acquirea corrected captured image by correcting non-linear distortion in thecorrected image, wherein the tilt recognition unit recognizes the tiltwith an accelerometer.